Monte-Carlo Permutation Tests
نویسنده
چکیده
The distribution of running time could be anything at all. We didnt even say whether it is continuous or discrete. Even so, the null hypothesis does induce a probability distribution on the test statistic without any other information. Specifically, it implies that the samples are indistinguishable and exchangeable. Our test statistic is a mean of 6 values (algorithm B running times) minus the mean of 8 values (algorithm A running times). If the null hypothesis is true, these measurements are indistinguishable. So there is a test statistic distribution induced by all permutations of the 14 values with the first 8 labelled as “A” and the last 6 as “B”. The resulting test statistic distribution is called the permutation distribution. The permutation distribution has no simple analytic form, but it can be generated numerically by enumerating permutations and building a histogram. In practice the number of permutations is usually too large to enumerate. But we can approximate the permutation distribution to arbitrary accuracy by using enough random permutations. An approximate permutation distribution built from one million random permutations is shown in figure 1. Like any other test statistic distribution, we can use this one to determine the p-value of the test statistic on our original data. The actual difference in mean times between algorithms A and B is 13.0. When we plot this on the test statistic distribution, we find that the fraction of permutations of the original data which gave an equal or higher test value was 0.0188 (the two-sided p-value is 0.0376). This p-value is remarkably close to the analytic value we derived for hypothesis 2 using a t-distribution. The permutation test above is an example of a non-parametric test. We made no assumption about the distribution under the null hypothesis. Usually, when you weaken the assumptions in the null hypothesis it becomes harder to reject as our earlier example showed. But the permutation test rivaled the sensitivity of a parametric t-test assuming equal variances. This wasnt a fluke. Permutation tests often rival or even exceed the performance of parametric tests. They are remarkably simple to design and implement the only “hard” part is generating enough random permutations to get the desired accuracy. But on today’s machines the computing time needed is very small.
منابع مشابه
Fast Permutation Tests that Maximize Power Under Conventional Monte Carlo Sampling for Pairwise and Multiple Comparisons
متن کامل
A Gentle Introduction to Resampling Techniques
2 Overview of resampling 2 Permutation Methods 3 Bootstrapping 3 Monte Carlo 4 Failure of t-test: Girl Scout cookie sales 4 Demonstration of permutations 5 Demonstration of Howell‟s resampling programs 5 Howell‟s example: Hurry up, Fella! 5 parametric t 6 Permutation tests (also called Randomization tests) 6 t as a measure of an effect 6 Howell‟s Resampling program 7 Howell‟s Randomization (per...
متن کاملOn Small Sample Properties of Permutation Tests: a Significance Test for Regression Models*
In this paper, we consider a nonparametric permutation test on the correlation coefficient, which is applied to a significance test on regression coefficients. Because the permutation test is very computerintensive, there are few studies on small-sample properties, although we have numerous studies on asymptotic properties with regard to various aspects. In this paper, we aim to compare the per...
متن کاملComparative Power Of The Independent t, Permutation t, and WilcoxonTests
The nonparametric Wilcoxon Rank Sum (also known as the Mann-Whitney U) and the permutation t-tests are robust with respect to Type I error for departures from population normality, and both are powerful alternatives to the independent samples Student's t-test for detecting shift in location. The question remains regarding their comparative statistical power for small samples, particularly for n...
متن کاملCovariate-adjusted nonparametric analysis of magnetic resonance images using Markov chain Monte Carlo.
Permutation tests are useful for drawing inferences from imaging data because of their flexibility and ability to capture features of the brain under minimal assumptions. However, most implementations of permutation tests ignore important confounding covariates. To employ covariate control in a nonparametric setting we have developed a Markov chain Monte Carlo (MCMC) algorithm for conditional p...
متن کاملExact and Asymptotically Robust Permutation Tests
Given independent samples from P and Q, two-sample permutation tests allow one to construct exact level tests when the null hypothesis is P = Q. On the other hand, when comparing or testing particular parameters θ of P and Q, such as their means or medians, permutation tests need not be level α, or even approximately level α in large samples. Under very weak assumptions for comparing estimators...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2008